Least Squares Temporal Difference Learning and Galerkin’s Method

The problem of estimating the value function underlying a Markovian reward process is considered. As it is well known, the value function underlying a Markovian reward process satisfied a linear fixed point equation. One approach to learning the value function from finite data is to find a good approximation to the value function in a given (linear) subspace of the space of value functions. We review some of the issues that arise when following this approach, as well as some results that characterize the finite-sample performance of some of the algorithms. 1 Markovian Reward Processes Let X be a measurable space and consider a stochastic process (X0, R1, X1, R2, X2, . . .), where Xt ∈ X and Rt+1 ∈ R, t = 0, 1, 2, . . .. The process is called a Markovian Reward process if • (X0, X1, . . .) is a Markov process, and • for any t ≥ 0, given Xt, Xt+1 the distribution of Rt+1 is independent of the history of the process. Here, Xt is called the state of the system at time t, while Rt+1 is the reward associated to transitioning from Xt to Xt+1. We shall denote by P the Markovian kernel underlying the process: Thus, the distribution of (Xt+1, Rt+1) given Xt is given by P(·, ·|Xt), t = 0, 1, . . .. Fix the so-called discount factor 0 ≤ γ ≤ 1 and define the (total discounted) return associated to the process R = ∞ ∑